Leadership dynamics in musical groups: Quantifying effects of musical structure on directionality of influence in concert performance videos.

Music ensemble performance provides an ecologically valid context for investigating leadership dynamics in small group interactions. Musical texture, specifically the relative salience of simultaneously sounding ensemble parts, is a feature that can potentially alter leadership dynamics by introducing hierarchical relationships between individual parts. The present study extended previous work on quantifying interpersonal coupling in musical ensembles by examining the relationship between musical texture and leader-follower relations, operationalised as directionality of influence between co-performers' body motion in concert video recordings. It was hypothesised that the directionality of influence, indexed by Granger Causality, would be greater for 'homophonic' textures with a clear distinction between melody and accompaniment parts than for 'polyphonic' textures with less distinction between melody and accompaniment. This hypothesis was tested by using pose estimation algorithms to track instrumentalists' body movements in a string quartet and a clarinet quintet, and then applying Granger Causality analysis to their head motion to estimate directional influence between instrumentalist pairs for sections of the pieces that varied in texture. It was found that Granger Causality values were generally higher (indicating greater directionality of influence) for homophonic than polyphonic textures. Furthermore, considering melody and accompaniment instrument roles revealed more evidence for the melody instrument influencing accompanying instruments than vice versa, plus a high degree of directionality among accompanying instruments, in homophonic textures. These observed patterns of directional information flow in co-performer body motion are consistent with changing leader-follower relations depending on hierarchical relations between ensemble parts in terms of the relative salience of melodic material in the musical texture. The finding that automatic pose estimation can detect modulations of leadership dynamics in standard video recordings under naturalistic performance conditions has implications for investigating interpersonal coordination in large-scale music video datasets representing different cultural traditions, and for exploring nonverbal communication in group activities more generally.

What's on your plate? Collecting multimodal data to understand commensal behavior.

Eating is a fundamental part of human life and is, more than anything, a social activity. A new field, known as Computational Commensality has been created to computationally address various social aspects of food and eating. This paper illustrates a study on remote dining we conducted online in May 2021. To better understand this phenomenon, known as Digital Commensality, we recorded 11 pairs of friends sharing a meal online through a videoconferencing app. In the videos, participants consume a plate of pasta while chatting with a friend or a family member. After the remote dinner, participants were asked to fill in the Digital Commensality questionnaire, a validated questionnaire assessing the effects of remote commensal experiences, and provide their opinions on the shortcomings of currently available technologies. Besides presenting the study, the paper introduces the first Digital Commensality Data-set, containing videos, facial landmarks, quantitative and qualitative responses. After surveying multimodal data-sets and corpora that we could exploit to understand commensal behavior, we comment on the feasibility of using remote meals as a source to build data-sets to investigate commensal behavior. Finally, we explore possible future research directions emerging from our results.

Interpersonal sensorimotor communication shapes intrapersonal coordination in a musical ensemble.

Social behaviors rely on the coordination of multiple effectors within one's own body as well as between the interacting bodies. However, little is known about how coupling at the interpersonal level impacts coordination among body parts at the intrapersonal level, especially in ecological, complex, situations. Here, we perturbed interpersonal sensorimotor communication in violin players of an orchestra and investigated how this impacted musicians' intrapersonal movements coordination. More precisely, first section violinists were asked to turn their back to the conductor and to face the second section of violinists, who still faced the conductor. Motion capture of head and bow kinematics showed that altering the usual interpersonal coupling scheme increased intrapersonal coordination. Our perturbation also induced smaller yet more complex head movements, which spanned multiple, faster timescales that closely matched the metrical levels of the musical score. Importantly, perturbation differentially increased intrapersonal coordination across these timescales. We interpret this behavioral shift as a sensorimotor strategy that exploits periodical movements to effectively tune sensory processing in time and allows coping with the disruption in the interpersonal coupling scheme. As such, head movements, which are usually deemed to fulfill communicative functions, may possibly be adapted to help regulate own performance in time.

Multisensory Perception and Learning: Linking Pedagogy, Psychophysics, and Human-Computer Interaction.

In this review, we discuss how specific sensory channels can mediate the learning of properties of the environment. In recent years, schools have increasingly been using multisensory technology for teaching. However, it still needs to be sufficiently grounded in neuroscientific and pedagogical evidence. Researchers have recently renewed understanding around the role of communication between sensory modalities during development. In the current review, we outline four principles that will aid technological development based on theoretical models of multisensory development and embodiment to foster in-depth, perceptual, and conceptual learning of mathematics. We also discuss how a multidisciplinary approach offers a unique contribution to development of new practical solutions for learning in school. Scientists, engineers, and pedagogical experts offer their interdisciplinary points of view on this topic. At the end of the review, we present our results, showing that one can use multiple sensory inputs and sensorimotor associations in multisensory technology to improve the discrimination of angles, but also possibly for educational purposes. Finally, we present an application, the 'RobotAngle' developed for primary (i.e., elementary) school children, which uses sounds and body movements to learn about angles.

The influence of yaw rotation on spatial navigation during development.

Sensory cues enable navigation through space, as they inform us about movement properties, such as the amount of travelled distance and the heading direction. In this study, we focused on the ability to spatially update one's position when only proprioceptive and vestibular information is available. We aimed to investigate the effect of yaw rotation on path integration across development in the absence of visual feedback. To this end, we utilized the triangle completion task: participants were guided through two legs of a triangle and asked to close the shape by walking along its third imagined leg. To test the influence of yaw rotation across development, we tested children between 6 and 11 years old (y.o.) and adults on their perceptions of angles of different degrees. Our results demonstrated that the amount of turn while executing the angle influences performance at all ages, and in some aspects, also interacted with age. Indeed, whilst adults seemed to adjust their heading towards the end of their walked path, younger children took less advantage of this strategy. The amount of disorientation the path induced also affected participants' full maturational ability to spatially navigate with no visual feedback. Increasing induced disorientation required children to be older to reach adult-level performance. Overall, these results provide novel insights on the maturation of spatial navigation-related processes.

Author Correction: Multi-layer adaptation of group coordination in musical ensembles.

An amendment to this paper has been published and can be accessed via a link at the top of the paper.

Multisensory Interactive Technologies for Primary Education: From Science to Technology.

While technology is increasingly used in the classroom, we observe at the same time that making teachers and students accept it is more difficult than expected. In this work, we focus on multisensory technologies and we argue that the intersection between current challenges in pedagogical practices and recent scientific evidence opens novel opportunities for these technologies to bring a significant benefit to the learning process. In our view, multisensory technologies are ideal for effectively supporting an embodied and enactive pedagogical approach exploiting the best-suited sensory modality to teach a concept at school. This represents a great opportunity for designing technologies, which are both grounded on robust scientific evidence and tailored to the actual needs of teachers and students. Based on our experience in technology-enhanced learning projects, we propose six golden rules we deem important for catching this opportunity and fully exploiting it.

Multi-layer adaptation of group coordination in musical ensembles.

Group coordination passes through an efficient integration of multimodal sources of information. This study examines complex non-verbal communication by recording movement kinematics from conductors and two sections of violinists of an orchestra adapting to a perturbation affecting their normal pattern of sensorimotor communication (rotation of half a turn of the first violinists' section). We show that different coordination signals are channeled through ancillary (head kinematics) and instrumental movements (bow kinematics). Each one of them affect coordination either at the inter-group or intra-group levels, therefore tapping into different modes of cooperation: complementary versus imitative coordination. Our study suggests that the co-regulation of group behavior is based on the exchange of information across several layers, each one of them tuned to carry specific coordinative signals. Multi-layer sensorimotor communication may be the key musicians and, more generally humans, use to flexibly communicate between each other in interactive sensorimotor tasks.

Computational Commensality: From Theories to Computational Models for Social Food Preparation and Consumption in HCI.

Food and eating are inherently social activities taking place, for example, around the dining table at home, in restaurants, or in public spaces. Enjoying eating with others, often referred to as "commensality," positively affects mealtime in terms of, among other factors, food intake, food choice, and food satisfaction. In this paper we discuss the concept of "Computational Commensality," that is, technology which computationally addresses various social aspects of food and eating. In the past few years, Human-Computer Interaction started to address how interactive technologies can improve mealtimes. However, the main focus has been made so far on improving the individual's experience, rather than considering the inherently social nature of food consumption. In this survey, we first present research from the field of social psychology on the social relevance of Food- and Eating-related Activities (F&EA). Then, we review existing computational models and technologies that can contribute, in the near future, to achieving Computational Commensality. We also discuss the related research challenges and indicate future applications of such new technology that can potentially improve F&EA from the commensality perspective.

Measuring social interaction in music ensembles.

Music ensembles are an ideal test-bed for quantitative analysis of social interaction. Music is an inherently social activity, and music ensembles offer a broad variety of scenarios which are particularly suitable for investigation. Small ensembles, such as string quartets, are deemed a significant example of self-managed teams, where all musicians contribute equally to a task. In bigger ensembles, such as orchestras, the relationship between a leader (the conductor) and a group of followers (the musicians) clearly emerges. This paper presents an overview of recent research on social interaction in music ensembles with a particular focus on (i) studies from cognitive neuroscience; and (ii) studies adopting a computational approach for carrying out automatic quantitative analysis of ensemble music performances.